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The  Good  News 

Disciplined  process  improvement  can  lead  to  better 
program  performance 

•  Meeting  Schedule  and  cost  commitments 

•  Product  quality  and  fitness  for  use 

Many  examples  demonstrate  this  quantitatively 
&  quite  convincingly 

•  Presented  at  this  conference  and  elsewhere 

•  http://www.sei.cmu.edu/cmmi/results.htm 
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The  Not  So  Good  News 

Skepticism  remains 

•  About  the  value  of  investing  in  improved  process 
capability 

•  In  both  Systems  Engineering  &  Software 

Instances  exist  of  less  than  stellar  product  delivery 

•  By  high  maturity  organizations  as  well  as  low 

More  and  better  evidence  is  needed  to: 

•  Convince  others  who  are  not  us... 

•  &  support  evidentially  based  process  improvement 


©  2005  by  Carnegie  Mellon  University 


page  4 


Carnegie  Mellon 

Software  Engineering  Institute 


Often  Heard  “Answers” 

“Maturity  Levels  are  meaningless” 

“The  high-maturity  organizations  are  not  applying  high-maturity 
practices  to  these  unsuccessful  programs.” 

“Process  is  just  one  element  of  program  success.  The  program 
failures  may  arise  from  weaknesses  in  the  people  or  the 
technology  applied  to  the  project.” 

“A  low-maturity  acquirer  prevents  the  organization  from 
performing  at  a  high  maturity  level.” 

“The  programs  are  unprecedented,  and  the  required  technology 
is  not  available.” 

...  and  many  more 
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The  “Real”  Answer 
We  don’t  know ! 

Most  of  the  evidence  comes  from  case  studies 

•  Which  can  be  accused  of  “cherry  picking” 

-  Fairly  or  not 

•  Failures  are  rarely  reported  publicly 

•  Circumstances  differ 

-  The  results  can  be  very  instructive  in  some  instances 

-  But,  they  may  not  be  applicable  elsewhere 

More  &  different  kinds  of  evidence  are  needed 

•  To  support  good  business  &  engineering  decisions 

•  Of  course,  some  will  never  be  convinced... 
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What  Else  Is  Needed? 

Credible  comparative  evidence  is  sorely  needed 

•  Proactively  elicited  from  all  parties 

•  To  better  demonstrate  the  statistical  relationships 
between  process  capability  &  program  performance 

-  Controlling  for  other  characteristics  that  may  affect 
both 

•  Using  the  same  measures  to  benchmark: 

-  Process  capability 

-  Performance  outcomes 

-  Product  characteristics 

-  Other  pertinent  contextual  differences 
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What  Causes  Program  Failure? 

Are  invalid  maturity  level  appraisals  the  only  cause? 

There  are  many  other  possible  reasons 

•  Requirements  volatility 

•  Contract  revisions  &  non  contractual  scope  creep 

•  Criticality  and  complexity 

•  Lack  of  precedentedness  &  domain  experience 

•  New  &  unproven  technologies 

•  Maturity  level  mismatches  &  other  poor  relationships 
among  acquirers,  contractors  &  subcontractors 
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Measuring  Program  Costs  &  Benefits 

Broadly  applicable  quantification  of  costs  &  benefits 
remains  elusive 

•  Complicated  by  the  lack  of  a  broadly  accepted  definition 
of  Systems  Engineering 

•  Insufficient  identification  and  tracking  of  Systems 
Engineering  costs  &  efforts 

•  Exacerbated  by  increasing  complexity  &  size  of 
systems  &  Systems  of  Systems 
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Our  Approach 
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Purpose 

Initial  focus  on  demonstrating  the  effectiveness  of 
Systems  Engineering 

Also  allows  us  to  address  quantitatively: 

•  The  reasons  why  programs  from  high  maturity 
organizations  sometimes  fail 

•  The  likelihood  of  program  failure  as  a  function  of 
organizational  process  maturity 

A  Comprehensive  Survey 

•  Of  defense  contactors  &  subcontractors 

•  In  collaboration  with  NDIA  Systems  Engineering 
Division  to  reach  a  broad  constituency 
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Focus  on  Systems  Engineering 

Focus  on  industry  members  of  NDIA  that  are  prime 
contractors  &  subcontractors 

•  Collect  feedback  from  project  /  program  managers 

Worked  with  a  committee  of  respected  systems 
engineers  to: 

•  Come  to  agreement  on  a  workable  definition  of 
Systems  Engineering 

-  Not  an  easy  task? 

-  Agreed  early  to  focus  on  CMMI  processes 
...  without  encouragement  from  the  SEI 

•  Provide  domain  expertise  on  other  aspects  of  survey 
content 

•  Help  craft  &  implement  a  viable  sample  selection  plan 
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Finding  the  Answer 

This  survey  addresses  individual  programs 

•  It  assesses  key  SE  practices  used  on  those  programs 

-  The  assessed  practices  are  derived  from  the  CMMI 

•  It  collects  context  information  for  those  programs 

-  Acquirer  capabilities,  technological  difficulty, 
contractor  experience,  etc. 

•  It  collects  performance  metrics  on  those  programs 

Analysis  of  the  survey  data  will  enable  us  to  see 
correlations  between  program  performance  and: 

•  CMMI  practices  (individual  and  ensemble) 

•  Other  program  characteristics 
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Narrowing  the  Scope 


CMMI-SW/SE  vl.1 

•  22  Process  Areas 

•  157  Goals 

•  539  Practices 

•  402  Work  Products 


Systems 

Engineering 

Filter 


•  1 3  Process  Areas 

•  27  Goals 

•  75  Practices 
•185  Work  Products 


Size  Constraint 
Filter 


•  1 0  Process  Areas 

•  19  Goals 

•  34  Practices 

•  63  Work  Products 
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Eliciting  Accurate  &  Honest  Answers 

Can  be  difficult  to  elicit  sensitive  information  from 
defense  contractors 

Reticence  to: 

•  Disclose  proprietary  advantages 

•  Admit  weaknesses  publicly 

•  Compromise  future  business  opportunities 

Crucial  to  assure  (&  deliver)  strict  non  disclosure  of 
all  information  provided 
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A  Promise  of  Anonymity 

To  elicit  honest  answers  without: 

•  Compromising  business  assets 

•  Threat  of  reprisal 

Necessary  for  the  survey  results  to  be  accurate  and 
useful  for  all  concerned 

•  Including  the  participating  organizations 

Survey  respondents  directed  to  a  web  portal 

•  Obtain  a  randomly  assigned  URL 

•  Known  neither  to  the  SEI  or  their  own  management 
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Sample  Selection  &  Implementation 

Committee  members 

•  Contact  representatives  of  key  organizations  to  request 
their  participation  in  the  survey 

•  Remind  them  to  have  their  people  complete  the  survey 

Organizational  points  of  contact 

•  Obtain  needed  commitment  from  senior  management 

•  Choose  survey  respondents  without  regard  to  program 
success 

•  Remind  the  respondents  to  complete  their  forms  on  a 
timely  basis 
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Step  6: 


©  2005  by  Carnegie  Mellon  University 


page  18 


Carnegie  Mellon 

Software  Engineering  Institute 


The  Survey  Instrument 


Self-administered 

•  Formatted  for  web-based 
deployment 

•  Option  for  off-line  completion 

Confidentiality 

•  No  elicitation  of  identifying  data 

•  Anonymous  response  collection 

•  Responses  accessible  only  to 
authorized  SEI  staff 


Section  1 

Context 

(Program  Characterization) 

Section  2 

Process  Capability 

(Systems  Engineering 
Evidence) 

Section  3 


Integrity 

•  Data  used  only  for  stated  purpose 

•  No  attempt  to  extract  identification 
data 

Self-checking 


Project  /  Program 
Performance 
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Contextual  Measures  Include 


Product  characteristics 
Contractual  obligations 
Project  context 
Organizational  context 


Section  1:  Characterization 


The  objective  of  this  section  is  to  gather  information  to  characterize  the  project  under 
consideration.  This  information  will  assist  the  survey  analysts  in  categorizing  the  project, 
and  the  executing  organization  to  better  understand  your  responses. 


1.1 


1.1.1 


Project  -  information  to  characterize  the  specific  project  under  discussion. 
Size,  stability,  lifecycle  phase,  subcontracting,  and  application  domain  are 
among  the  parameters  used  for  program  characterization^ 


What  phases  of  the  integrated  product  lifecycl^, 
comprise  this  project  (check  all  that  apply),  jmd 
what  phase  are  you  presently  executing  (cln 


tended  in  project 
k  all  that  apply 

Current 
phase 
(check  1) 

Concept  Refinement 

Technology 

Development  and 

Demonstration 

Development 

Manufacturing 

Verification 

Training 

Deployment 

Operation 

Support 

Disposal 


1.1.2 

What  is  the  current  total  contract  value  (US$)  of 
your  project? 

$ 

1.1.3 

What  was  the  initial  contract  value  (US$)  of  your 
project? 

$ 

1.1.4 

How  many  contract  change  orders  have  been 
received? 
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Process  Capability 

Process  definition 
Project  /program  planning 
Risk  management 
Requirements  development 
Requirements  management 
Trade  studies 
Interfaces 
Product  structure 
Product  integration 
Test  and  verification 
Project  /  program  reviews 
Validation 

Configuration  management 


Section  2:  Systems  Engineering  Evidence 


Rate  your  agreement  with  the  following  statements 


2.1 


Process  Definition 


2.1.1  This  project  utilizes 
engineerin$pricespi 
the'TjFoiect 


2.2 


2.2.1 


[his  prp 
Laicij 

i  u]W)  ork 

3r£2Kdown 
Structure  (WBS) 
that . . . 


a.  .  includes  task  descriptions  and 
work  package  descriptions 


b.  ...  is  based  upon  the  product 
structure 


c.  . .  .is  developed  with  the  active 
participation  of  those  who 
perform  the  systems  engineering 
activities 


r  r  r  r 


r  r  r  r 


r  r  r  r 
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Program  Performance 


Uses  measures  common 
to  many  organizations 

•  Earned  Value 

•  Award  Fees 

•  Technical 
Requirements 
Satisfaction 

•  Milestone  Satisfaction 

•  Problem  Reports 


Section  3:  Project  Performance  Metrics 


3.1  Earned  Value  Management  System  (EVMS) 


Rate  your  agreement  with  the 
following  statements 


m  a  Timely  manner  (i.e. 
^rffwithin  2  weeks)? 


3.1.3  The  requirement  to  track  and  report 
EVMS  data  is  levied  upon  the 
project’s  suppliers. 


3.1.4  Variance  thresholds  for  CPI  and  SPI 
variance  are  defined,  documented, 
and  used  to  determine  when 


©  2005  by  Carnegie  Mellon  University 


page  22 


Carnegie  Mellon 

Software  Engineering  Institute 


What’s  Next? 
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Survey  Status 

Survey  instrument  development  complete 

•  Web  deployment  complete 

•  Pretest  in  progress 

Respondent  identification  in  progress 
Response  collection  through  early  February 
Data  analysis  and  report  by  2Q  CY2006 
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Risks 

Respondent  selection  takes  longer  than  planned 

Response  rate  is  too  low  to  provide  confidence  in 
generalizability 

•  The  committee  liaisons  &  organization  focal  points  of 
contact  need  to  remind  people  to  reply 

Respondent  selection  or  survey  responses  will  be 
biased 

•  May  need  to  allow  more  time  for  people  to  reply 

-  To  avoid  excluding  the  busiest  people  and  at-risk 
projects 

•  Crucial  for  senior  management  to  encourage  honest  & 
forthright  answers 
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How  Can  You  Help? 

Agree  to  have  your  organization  participate  if  you  are 
contacted  by  a  committee  member 

•  Select  respondents  without  regard  to  their  program 
success 

•  Provide  encouragement,  &  resources,  for  the 
respondents  to  complete  their  surveys 

-  Honestly  &  openly 

-  Without  fear  of  reprisal 

Encourage  others  to  participate 

•  As  potential  respondents  &  in  the  respondent  selection 
itself 
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Contact  Information 


Joseph  P.  Elm 
ielm@sei.cmu.edu 

Software  Engineering  Institute 
Carnegie  Mellon  University 

Dennis  Goldenson 
dg@sei.cmu.edu 

Pittsburgh,  PA  15213-3890 
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